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Abstract. Recently, a rising need to understand (novice) users of digital cartographic products 
has surfaced: how do they interpret and process the visual information, and how is this retrieved 
and used later on? A user study was conducted that combined several techniques to investigate 
these cognitive processes: eye tracking, thinking aloud and sketch maps. I n total, 24 participants 
took part in the study, from which 12 were experts in the cartography. All participants had to 
study (learn) the content of four different topographic maps depicted on a screen. After studying 
each map, the participants were instructed to draw this map from memory (retrieve infor- 
mation), using paper and pencils. The results indicate that both user groups address the same 
cognitive processes, but these are positively influence by the expertise and background 
knowl edge of the experts. 
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1 Introduction 

1.1 Background 

Although cartographic products have evolved drastically over time, their main function still re- 
mains the: communication of geographic information. However, the possibilities and limitations 
inherently linked with screen displays have an impact on how maps can be presented to the us- 
er, and thus on how the information is perceived and thus used later on (MacEachren, 1995; 
Matlin, 2002). As a consequence, there is a rising need to understand the map user: how does he 
interpret and process the visual information on the maps, and how is this retrieved and used 
later on (e.g. Fabrikant & Lobben, 2009; Harrower, 2007; Montello, 2002, 2009; Slocum et al., 
2001). The users' level of expertise (knowledge already stored in memory) could also have a sig- 
nificant influence of this (e.g. Gilhoolyetal., 1998; Hambrick&Engle, 2002). 

1.2 Studying the map users' cognitive processes 

Using eye tracking, the position wherea user is looking (Point of Regard, POR) is registered at a 
certain sampling rate. This gives insights in the user's attentive behaviour, which is an essential 
step in object recognition and thus in the interpretation process. Besides the position of the 
POR, other usable metrics (such as fixation duration) can be derived from the eye movement 
data, which are closely linked with the user's cognitive processes (Duchowski, 2007; J acob & 
Karn, 2003; Poole& Ball, 2006). 



Other methods are more suitable to investigate the retrieval of information from memory. Thin- 
king aloud is, as such, a well integrated method in psychological research. During a thinking 
aloud study, participants are typically asked to say out loud every thought that comes into mind. 
The verbal protocols obtained during concurrent verbalisation correspond to the content of the 
working memory at that moment. As a consequence, these verbal protocols are direct and unfil- 
tered (Nielsen, 1993; Trickett &Trafton, 2007; van Someren et al., 1994). Furthermore, sketch 
maps could also be valuable tools to study the users cognitive processes. Huynh and Doherty 
(2007, p.286) defined sketch maps as "the extraction of information from a mental map through 
drawing". 



2 Study Design 

2.1 Participants 

Two groups of participants were selected to take part in the study, containing each 12 persons, 
with an equal share of males and females. The first group consisted out of experts in map use 
and cartography. All participants in this group had at least a Master degree in Geography or Ge- 
omatics and were employed at the Department of Geography at Ghent University. The second 
group consisted out of participants who did not receive any previous cartographic training and 
did not work with maps on a professional level. 

2.2 Task and Stimuli 

The participant was told he had to remember the general structure (e.g. roads, rivers, forests, 
etc.) of the map that would be shown on a screen. No time limit was set on this task to avoid 
influence from time pressure. While executing this task, the participant's eye movements were 
registered. After studying the map, the participant took place at another table with a blank A3 
paper, colour pencils, a gum, and a headset. During this second phase of the test, the participant 
had to draw the map he had just studied from memory. While drawing, the participant was in- 
structed to think out loud'. Also in this case, no time limit was set to avoid biases due to time 
pressure. Finally, at the end of the experiment the participants had to fill in a questionnaire to 
obtain background information: age, sex, experience, familiarity with thestimuli, etc. 

This task of remembering and drawing a map was repeated for all four stimuli presented on a 
screen (see Figure 1). These are based on topographic maps on 1: 10 000 created by the Belgian 
national mapping agency (I GN-NGI). Thestimuli were presented i n the same order to the par- 
ticipants. Only half of the participants (within each group) saw Map3a, the other half the mir- 
rored version Map3b. Consequently, the influence of recognizing the map content is eliminated. 
This is not the case for the mirrored maps Mapl and Map4, which were displayed to all partici- 
pants. Finally, the colours of Map2 are adapted: the water bodies are depicted in red instead of 
cyan; the background colour of the villages is presented in purple instead of yellow. These adap- 
tations in the map image could also influence the user's cognitive processes while reading, inter- 
pret ng or retri evi ng the map's content. 
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Figure L stimuli that were used during the test 



2.3 Apparatus and Recordings 

The user study took place at the Eye Tracking Laboratory of the Department of Experimental 
Psychology (Ghent University). This lab is equipped with an EyeLinklOO from SR Research. The 
stimuli were depicted on a 21inch monitor. The software DataVi ewer (SR Research) was used 
to aggregate the raw data into meaningful measurements, such as fixations. Detailed infor- 
mation regarding these metrics can be obtained for indicated Areas of I nterest (AOIs): the num- 
ber of fixations, the total dwell time and the average duration of the fixations. 

Two cameras and a headset were used to record the auditory and visual components for the 
thinking aloud study. The headset was connected to a laptop which webcam recorded the front 
view of each participant: face and A3 paper on which they were drawi ng. This was an Acer Crys- 
tal Eye Webcam that captured HD videos, including the audio recorded with the headset. 

A second H D camera (Sony HandyCam H DR-CSI15E) was placed on a tripod and captured the 
drawing process on the A3 page from a top view. The coloured post-it placed next to each A3 
page was visible in all recorded videos. Furthermore, the code assigned to each participant was 
written on the post-its and could thus be read from the top view videos (from the HandyCam). 
The synchronisation of the recordings from the two cameras (webcam and HandyCam) was 
based on the moment in time when the post-it was placed. The software ELAN was used to 



manual ly synchronise the videos and prepare them to be able to analyse the derived verbal pro- 
tocols. 



3 Methodology and Results 
3.1 Studying the Map 

The registered eye movement data was analysed both quantitatively and qualitatively. This com- 
bination ensures to obtain the most complete view on the users' cognitive processes and the dif- 
ferences between the two user groups. 

3.1.1 Quantitative Analysis 

Table 1 lists the mean values (M) and standard deviations (SD) for the average fixation dura- 
tions, the number of fixations per second and the duration of the trial, both for the expert and 
novice users. The last column in this table shows the results of a one way ANOVA between the 
two user groups. From this table it can be derived that the expert users have significantly shorter 
and consequently more fixations than the novice users. These findings are in correspondence to 
the results of Ooms et al., 2012, where the eye movement metrics resulting from a visual search 
on a very basic map design were described. This indicates that experts can interpret the same 
map more efficiently than novices. 



Table L Statistical comparison (average fixation duration; number of fixations per second; duration of 
the trial) bet ween expert and novice map users 





Experts 


Novices 


ANOVA 




M 


SD 


M 


SD 


F P 


AvgFixDur(s) 


.308 


.046 


.339 


.062 


7.578 .007 


Fix/s 


2.804 


.467 


2.566 


.469 


6.235 .034 


TrialDur (s) 


335.7 


128.0 


205.7 


128.5 


10.516 .002 



3.1 .2 Qualitative Analysis: Gridded Visualisations 

A grid of 640 AOIs (40x32) was placed over each map image. In Figure 2, this grid of AOIs is 
depicted in yellow; the cyan circles represent the participant's fixations. The AOI -report from 
DataViewer was restructured as such that every AOI was located on its correct location on the 
map image. Furthermore, the obtained values (fixation count, total dwell time, average fixation 
duration) were normalised to correct for the varying trial durations (see also Table X bottom 
row). It was decided to use a uniform trial duration of 300 s (or 5 minutes), which corresponds 
cl osel y to the average over al I users. 

Next, a classification and colour ramp to effectively and objectively visualise the obtained data in 
the grid had to be chosen, taking into account the fixations' characteristics (eg. the link between 
the fixation count and dwell time) and using a colour scheme based on ColorBrewer (Brewer, 
2012) (see Table 2). Figure 3 shows examples of the obtained results, with a comparison be- 
tween both user groups: Mapland Map2 depict the average fixation count; Map3 and Map4 
depict the maximal fixation count. The same methodology can be applied to visualise the distri- 
bution of the users' average and maximum dwell time or fixation duration. The main advantage 



of using this gri doled visualisation, in comparison to the often used heatmaps, isthelevel of con- 
trol (and thus objectivity) with which thedata can be represented. 



439 ™ 




Figure 2. Grid of AOI s placed over each map image (yellow) to visually analyse the users' fixations (cyan) 



Table 2. Classification and colour scheme for fixation duration 

Variable Classification and colour schemes 



FixCount 


[0-U 


[l-2[ 


[2-4[ 


[4-6[ 


[6-8[ 


[8-10[ 


[]D-20[ 


[20-...] 


FixDur 


[.000- 
.325[ 


[.325- 
.650[ 


[.650- 
1300[ 


[1300- 
1950[ 


[1950- 
2.600[ 


[2.600- 
3.250[ 


[3.250- 
6.500[ 


[6.500- 

] 


Colour 
(RGB) 


255 


247 


217 


189 


150 


99 


37 
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Figure 3. Grid of AOI s depicting the average and maximal fixation count for experts (top row) and novic- 
es(bottom row) for the stimuli (left to right) 



Based on these 2D grids, statistical ANOVA tests between all corresponding cells of different 
grids can be executed. This method has been applied to the mirrored map equivalents: Maplvs. 
Map4 and Map3a vs. Map3b. On the top row of Figure 4, a comparison is made between the 
original data. On the bottom row, the registered eye movements of the mirrored maps (Map4 
and Map3b) are mirrored again and then compared with the original maps (Mapland Map3a). 
Dark cells indicate a high significant difference; white cells show no significant difference. This 
allows investigating 'how weir the registered eye movement data is reflecting the main structur- 
ing el ements on the map. 




Figure 4. Statistical comparison of the number of fixation between two mirrored maps 
3.1.3 Discussion 

When combining the obtained results, similar patterns in all users' attentive behaviour can be 
discovered. They focus on the main (linear) structuring elements on the map, such as rivers and 
road. These elements are clearly reflected in the 2D gridded visualisations, on the one hand, but 
significant proof can be found in the statistical grids. Thetwo grids on top of Figure4 (the origi- 
nal grids) show a high number of dark cells, whereas the grids at the bottom (re-mirrored data) 
show hi gh correspondi ng val ues. 

Nevertheless, the statistical tests also indicate a significant difference in how experts and novic- 
es interpret the data. Experts are significantly more efficient in their attentive behaviour (less 
distracted by deviating colours, familiar objects or less important elements) and interpretation 
process (shorter fixati ons) . 

3.2 Thinking Out Loud and Sketched Maps 

While drawing, the users had to say out loud every thought that came into mind. These verbali- 
sations are closely linked to the active content of the participants' working memory and thus to 
the cognitive processes and strategies needed to retrieve the information from (the long term) 
memory. The obtained verbal protocols were segmented in two ways to contribute to the validity 
of the results. The sketched maps also provide important information regarding the users' in- 
formation retrieval processes. 

3.2.1 Counting and Comparing Word Use 

All words that were verbalized during the thinking aloud sessions were counted, ordered based 
on its frequency, and compared between both user groups. Not surprisingly, the words that are 
most frequently used, correspond a lot between both user groups as these are very general 
words, I ike the', 'it', 'a', 'has', etc. However, when investigating the words that are I ess frequently 
used (eg. 30-50% of the ordered list), a clear distinction is noticed. Novices use more descrip- 



tive terms for the shape, colour and location of the map objects (e.g. line, red, middle, circle, 
etc.) , whereas the experts used their correct names (meander, major road, north, field, etc.). 



3.2.2 Analyzing the 'Full Thought' 

The participants' verbalised full thoughts (sentences) were analysed according to the scheme 
depicted in Figure 5. A psychological model was developed, based on which a coding scheme 
constructed that has four levels: map level, item level, confidence level, and actions (seeTable3 
to Table 6). Four codes were thus assigned to each of the segments to obtain the coded proto- 
cols. The duration of each full thought (and its associated code) is expressed as a ratio between 
zero and one, relative to the full length of the thinking aloud session. 
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Figure 5. Analysing verbal protocols obtained from a thinking aloud study (after van Someren et al, 1994) 



Table 3. Proposed codes related to the map level (Level 1) 



psych, model 


category 


code 


description 


Orientate 


orientate 


orl 


before the start of the task execution, gather thoughts, 
check tools, assignment, ... 


Execute Assignment 


execute 


exl 


drawing, talking, explaining, correcting items, ... 


Evaluate Result 


evaluate 


evl 


check if everything is on the map (count objects, check 
size, relative positions,...); not the correction of individual 
items when drawing them 


OK? 


evaluate 


evl 



Table 4. Proposed codes related to the item level (Level 2) 



psych, model 


category 


code 


description 


Consult WM 


gather thoughts 


gt2 


participant is clearly thinking, gathering thoughts, finding 
links between items, consult previous 
knowledge, similarity between objects 


Next Item 


gather thoughts 


gt2 


LTM to WM 


gather thoughts 


gt2 


Draw 1 tern 


draw 


dw2 


items aredraw on paper, colour pencil was taken, ... 


Evaluate Item 


evaluate 


ev2 


check correctness of item that was drawn, or items 
related to it 


OK? 


evaluate 


ev2 


Adapt 1 tern 


correct 


cr2 


correct items which wereevaluated (drawing, erasing) 



Table 5. Proposed codes related to the partici pants' level of confidence (Level 3) 



confidence 


category 


code 


description 


Confident 


confident 


cf3 


participant issurethat it iscorrect 


Neutral 


neutral 


nt3 


level of confidence is not expressed 


Not Confident 


not confident 


nc3 


participant is not sure that it is correct 



Table 6. Proposed codes related to the partici pants' actions (Level 4) 



action 


category 


code 


description 


Draw 


draw 


dw4 


participant draws an item 


Take Penci 1 


take pencil 


tp4 


parti ci pant takes a penci 1 


Correct 


correct 


cr4 


participant corrects item that was drawn previously 


Erase 


erase 


er4 


participant uses the eraser 


Fill Colour 


fill colour 


fc4 


participant isfilling up items with colour 


Checking 


checking 


ck4 


parti ci pant checks draw i ng 


Talking 


talking 


tk4 


participant is only talking 



The graphs in Figure 6 show a very similar structure for the experts and the novices. This indi- 
cates that the same cognitive processes were addressed while solving this problem, which is in 
this case rememberi ng and drawing the i nformation from a map that was studied previously. 
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Figure 6. Overview of the segmented protocols (full thought) per level and map 



3.2.3 Sketched Maps 

The maps that were drawn by the participants were compared with the ori- 
ginal screen maps. Based on how well these maps were drawn, a score was 
assigned to them. The aesthetics of the sketched objects or drawing skill of 
the participants were not taken into account. A similar scoring system as 
used by Kulhavy and Stock (1996) and Thorndyke and Stasz (1980) was 
appl i ed to obtai n a score for each map. I f the obj ect was present and drawn 
on the correct location a score of one point was assigned to it. If the object 
was present, but drawn on a considerably wrong location, only half a point 
was assigned to it. The results are presented in Figure 7. 




hydrography lantf cover settlements roads 




Figure 7. Overview of the scores (%) on the drawn maps, per theme (left) and per 
map (right) 



The order with which the different structuring elements on the map were 
drawn can give insights in how these elements were stored in the users' 
memory. Huynh et al. (2008) argue that this sequence gives insights in the 
hierarchical structure of the users' cognitive map: a basic framework in 
which lower level elements are subsequently placed. This hierarchical struc- 
ture i s cl osel y I i nked with the percei ved i mportance of obj ects by the user. 

For each map, eight categories of elements were identified (see Figure 8). 
These categories contain structuring map elements and are an extension of 
the four categories used to present the map scores: electrical wire (other), 
field (landcover), forest (landcover), settlement, hydrography, meadow 
(landcover), railroad (other), and road. A 'score' was assigned to each cate- 
gory based on the order with which the first element of that category was 
drawn: 100 for the first element, 50 for the second, 25 for the third, 5 for 
thefourth, 4 for the fifth, and so on. If a certain element was not drawn, no 
score was assigned. The three categories that are associated with the hig- 
hest scores are the forests, roads and hydrography. 




Figure 8. The order with which the objects were drawn for each map 



3.2.4 Discussion 

Using thinking aloud and sketch maps, it can be discovered how users 
structure the information in their memory (previously gathered from a digi- 
tal map). The protocol analyses on the full thoughts indicate that both user 
groups address the same cognitive processes to retrieve the information. 
However, a significant influence of expertise (amount of background 
knowledge) was found regarding how the retrieved information is struc- 
tured and consequently how much information could be retrieved. Experts 
could retrieve much more information, because they could link the newly 
gathered data with information that was already present in their memory. 
Because of the available background information, they actually have to re- 
member less and can structure is more efficiently. Consequently, it is better 
accessible during information retrieval. Novices, on the other hand, lack 
this background information. All perceived information has to be stored, in 
a rather unstructured, descriptive way making it harder to retrieve later on. 



4 Conclusion 



By combining different methods, it was possible to the study complete car- 
tographic communication process. Attentive behaviour and interpretation 
was investigated through quantitative and qualitative analyses of eye track- 
ing data, but also how much information was remembered and how this is 
structured in memory. The latter elements were studied using thinking 
aloud and sketch maps. 

1 1 was found that the mai n cognitive strategi es that were addressed are very 
similar for both expert and novice users. While studying the map, they fo- 
cus their attention on the main structuring elements on the map to create a 
reference frame in which all other elements can be placed. However, ex- 
perts can interpret the content significantly more efficiently and are less 
distracted (by deviating colours, familiar objects, less important objects, 
etc.). As a consequence of this and in combination with extensive back- 
ground knowledge, the experts could structure the i nterpreted information 
more logically through links with the existing information in memory. This 
al so al I ows easi er retri eval of the i nf ormati on . 

1 1 is thus i mportant that experts i n cartography, when creati ng cartographic 
products, are aware of the novice map users' cognitive limits. The visualisa- 
tion of the map elements should be keyed to the novice map users cognitive 
structures and processes so that they can interpreted more efficiently and 
effectively. This would facilitate the storage of the information in memory, 
maki ng easi er accessi bl e duri ng i nf ormati on retri eval . 
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